Some quick cleaning

The dataset came from Kaggle but was still a little messy, not sure why the artists column came wrapped in [‘name’]. Additionally some date formatting was inconsistent (as always), I imputed -01-01 if there was no date after the year.

df2 <- df %>% mutate(artists = str_remove_all(artists, "\\['"),
                     artists = str_remove_all(artists, "\\']"),
                     artists = gsub(",", " and ", artists),
                     artists = str_remove_all(artists, "^'|'$"), # work on this a bit more
                     decade = as.factor(floor(year/10)*10),
                     year = as.factor(year))
# Also would be cool to try to recognize gender by name and make a dummy column

Looking At the Data Using Lares

df_str(df, return = "plot")

Look at percentages and cumulatives, looks like no popularity is very common!

df %>% freqs(popularity, plot = T)

There is about the same amount of music every year in this dataframe

df %>% freqs(year, plot = T)

Basically newer stuff is more popular with little to no exceptions

df %>% mutate(popularity = round(popularity/10)) %>% freqs(popularity, year, plot = T, results = F, top = 50)

Looks like explicitness has a normal distribution compared to popularity

df %>% 
  mutate(explicit = as.factor(explicit),
         popularity = round(popularity/10)) %>%
  freqs(popularity, explicit, plot = T,
        title = "Popularity by Explicitness",
        subtitle = paste("Duncan Gates", Sys.Date()),
        results = F) 

Now we check out the distribution, there’s some really cool stuff here

df %>% distr(popularity, valence) # Some really cool density plots can be done here

There’s also some really long songs out there…

df %>% distr(duration_ms)

This looks more like the actual distribution

df %>% 
  filter(duration_ms < 400000) %>%
  distr(duration_ms)

Very interesting distribution here

df %>% distr(popularity)

Looks like things are a lot more explicit in 2000-2020 as one might expect, would be interesting to see how when this starts, or what drives it. I also wonder what happened in 1920-1940?

df %>%
  mutate(explicit = as.factor(explicit),
         new_era = ifelse(year %in% c(2000:2020), 1, 0)) %>%
  distr(explicit, new_era)

df %>% 
  mutate(decade = floor(year/10)*10) %>%
  distr(explicit, decade, custom_colours = T, abc = T) 

By the way mode is just whether the song is major or minor.

df %>% distr(mode, force = "char") 

You can even use ggplot2!

df %>%
  distr(popularity, loudness) + geom_point(color = "yellow")

Wouldn’t be data science without some random regressions, even more data science/machine learningy since the second one is a log odds table!

df %>%
  select(-c(id, name, artists, year, release_date, key)) %>%
  corr_cross(top = 10) # Look at top 10 correlations in the data, key messes with this idk why

table <- df %>%
  select(-c(id, name, artists, year, release_date, key)) %>%
  corr_var(popularity, logs = T, plot = F, top = 10) 
table %>% mutate(corr = kableExtra::cell_spec(corr, "html", color = ifelse(corr > 0, "blue", "red"))) %>%
  kableExtra::kable(format = "html", escape = F) %>% 
  kableExtra::kable_styling("striped", full_width = F, position = "center")

# wow OHSE is pretty dope check this out with a better dataframe

Fun with Machine Learning